Sorry for the long silence on this, but your thorough explanation on this has led me down quite a rabbit hole.

In short: you're absolutely right. You've convinced me that "normalizing" the manifest is absolutely necessary when we install it into the workspace. Further, I now that believe it extends beyond "library" crates and applies to all participants in a workspace, even those that only build binaries.

I have also discovered (I believe) that it isn't possible to patch a dependency that refers to a sibling a package's workspace. At first glance, it might seem okay to forego the patch and let cargo consume the dependency directly from the source directory, but that could result in unexpected behavior when a downstream build in a separate workspace tries to use that same dependency and consumes it from the install space instead. Ideally a developer would always re-build a package (i.e. the upstream dependency) which has been modified, but I could see someone distributing a built workspace that "worked on their machine" not knowing that the sources for the dependency package hadn't been properly rebuilt (and reinstalled).

I'm experimenting with code which unconditionally normalizes the manifest before operating on it. Maybe we can get away with only doing it for workspace packages, but until we can really enumerate everything that happens during cargo's manifest normalization process, I'm inclined to just do it every time.

The only supported way to normalize the manifest, as far as I can tell, is using cargo package. We can pass lots of flags to make this as lightweight as possible (like --no-verify --exclude-lockfile --offline), but it's still a pretty heavy hammer just to process the manifest. Maybe this would be a good feature request for cargo.

To use the manifest during our operations (namely build and install), I tried leveraging the --manifest-file option, but it appears that cargo looks for source files based on the location of the manifest and not the working directory so given that we don't want to change the package sources with the normalized manifest, it would mean that we need to populate the normalized manifest directory with the source files as well. We could extract all of the crate contents to do this, or we could copy/symlink the source files from the package source directory to avoid the decompression cycle. I'll punt the specifics of this question for later discussion.

Since we're essentially re-homing the whole crate at this point, the thought occurred to me that maybe we should just build it there and not use --manifest-file at all. However, it appears that while source file discovery is rooted from the manifest location, the discovery of .cargo/config.toml happens based on the working directory. We currently respect any .cargo/config.toml files that the user has in their source tree as this maintains better parity with standalone cargo behavior.

It's worth noting that making cargo package part of all package builds does impose some new restrictions on the manifests. Notably, all dependencies must now include version constraints and can no longer have only a path to another crate. The dependencies can however have a version = '*', as that value is prohibited only on crates.io and not by cargo itself. Really, this new restriction only applies to non-library crates, and IMO is not a really a problem. Here's an example of a dependency that would no longer be valid:

My (current) conclusion

Colcon operates on sources on a per-package basis. Effort is made to isolate each package, sometimes to validate proper dependency declaration, and sometimes to ensure that the resulting install space can be used by downstream processes or "child" workspaces. In many ways this is in opposition to much of how the cargo tool consumes workspace members, so we should make an effort to "de-workspace" the packages using cargo's manifest normalization. This should (help) ensure that we aren't crossing the package boundary in cargo in opposition with colcon's efforts, and allows us to patch dependencies that are (or were previously) workspace siblings.

Dragons

1. There's an argument that cargo workspaces, even outside the context of the cargo tool, violate colcon's package boundaries in their nature as information (namely metadata) from the workspace root affects the workspace member builds. I don't really see a way around this, and even if there was, circumventing workspaces entirely would cause a pretty heavy behavioral deviation from standalone cargo. I'm not even sure it would be possible to just "ignore" workspaces.
2. There's an argument that .cargo/config.toml discovery along the source directory chain is also a violation of colcon's package boundaries. This point is more interesting to me, and I'd love to hear thoughts from cargo users. While colcon isolates packages from other packages, it makes no effort (by design) to isolate package from the developer environment. If we moved the working directory into the "build" space, .cargo/config.toml discovery would still happen at the workspace root (at least for a "typical" workspace layout). Probably better to stay focused on what we need and revisit this later...
3. My proposal is clearly a further deviation from how standalone cargo operates under the hood, and I'm getting more and more concerned about build performance and integration with cargo-native IDEs and tooling. I won't pretend that I have any satisfying answers here. Moving away from consuming the source files directly in the source directories is a huge red flag in this context. My only hope would be that someday cargo might offer new features to better support our scenario.

Sorry lots of text there. I'll try to follow up with some prototypes and examples, and share any new discoveries along the way.